{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DAY5 离散特征的处理（独热编码）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "今天的任务分成以下几步\n",
    "1. 读取数据\n",
    "2. 找到所有离散特征\n",
    "3. 选择一个离散特征进行独热编码\n",
    "4. 采取循环对所有离散特征进行独热编码\n",
    "5. 加上昨天的内容 并且处理所有缺失值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 读取数据\n",
    "import pandas as pd\n",
    "data = pd.read_csv(r'E:\\study\\PythonStudy\\python60-days-challenge-master\\data.csv') # 此时data是一个DataFrame对象"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Id', 'Home Ownership', 'Annual Income', 'Years in current job',\n",
       "       'Tax Liens', 'Number of Open Accounts', 'Years of Credit History',\n",
       "       'Maximum Open Credit', 'Number of Credit Problems',\n",
       "       'Months since last delinquent', 'Bankruptcies', 'Purpose', 'Term',\n",
       "       'Current Loan Amount', 'Current Credit Balance', 'Monthly Debt',\n",
       "       'Credit Score', 'Credit Default'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# day4的课提到了 查看dataframe对象的列名，可以使用data.columns属性。\n",
    "data.columns "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Home Ownership\n",
      "Years in current job\n",
      "Purpose\n",
      "Term\n"
     ]
    }
   ],
   "source": [
    "# 打印所有的离散变量名\n",
    "# 在python中对于变量名常常用英文含义和下划线来命名，而不借助拼音，这是便于他人阅读和理解代码的一种习惯。\n",
    "# 连续的英文是continuous，离散的英文是discrete\n",
    "for discrete_features in data.columns:\n",
    "    if data[discrete_features].dtype == 'object':\n",
    "        print(discrete_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0            Own Home\n",
       "1            Own Home\n",
       "2       Home Mortgage\n",
       "3            Own Home\n",
       "4                Rent\n",
       "            ...      \n",
       "7495             Rent\n",
       "7496    Home Mortgage\n",
       "7497             Rent\n",
       "7498    Home Mortgage\n",
       "7499             Rent\n",
       "Name: Home Ownership, Length: 7500, dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 以Home Ownership为例，打印观察下\n",
    "data['Home Ownership']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Home Ownership\n",
       "Home Mortgage    3637\n",
       "Rent             3204\n",
       "Own Home          647\n",
       "Have Mortgage      12\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 需要进行编码，打印这个变量的值\n",
    "# vakue_counts()方法用于统计每个类别的个数，并返回一个Series对象。这个方法可以帮助我们快速了解数据集中每个类别的分布情况。\n",
    "data['Home Ownership'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Home Ownership：房屋所有权\n",
    "2. Rent：租房\n",
    "3. Own Home：拥有自有住房\n",
    "4. Have Mortgage：有抵押贷款\n",
    "\n",
    "可以发现并不具备顺序关系，因此可以采用one-hot编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Id', 'Annual Income', 'Years in current job', 'Tax Liens',\n",
       "       'Number of Open Accounts', 'Years of Credit History',\n",
       "       'Maximum Open Credit', 'Number of Credit Problems',\n",
       "       'Months since last delinquent', 'Bankruptcies', 'Purpose', 'Term',\n",
       "       'Current Loan Amount', 'Current Credit Balance', 'Monthly Debt',\n",
       "       'Credit Score', 'Credit Default', 'Home Ownership_Have Mortgage',\n",
       "       'Home Ownership_Home Mortgage', 'Home Ownership_Own Home',\n",
       "       'Home Ownership_Rent'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 对Home Ownership列进行独热编码\n",
    "data = pd.get_dummies(data, columns=['Home Ownership'])\n",
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到之前的Home Ownership已经被替换成了'Home Ownership_Have Mortgage','Home Ownership_Home Mortgage', 'Home Ownership_Own Home','Home Ownership_Rent'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Annual Income</th>\n",
       "      <th>Years in current job</th>\n",
       "      <th>Tax Liens</th>\n",
       "      <th>Number of Open Accounts</th>\n",
       "      <th>Years of Credit History</th>\n",
       "      <th>Maximum Open Credit</th>\n",
       "      <th>Number of Credit Problems</th>\n",
       "      <th>Months since last delinquent</th>\n",
       "      <th>Bankruptcies</th>\n",
       "      <th>...</th>\n",
       "      <th>Term</th>\n",
       "      <th>Current Loan Amount</th>\n",
       "      <th>Current Credit Balance</th>\n",
       "      <th>Monthly Debt</th>\n",
       "      <th>Credit Score</th>\n",
       "      <th>Credit Default</th>\n",
       "      <th>Home Ownership_Have Mortgage</th>\n",
       "      <th>Home Ownership_Home Mortgage</th>\n",
       "      <th>Home Ownership_Own Home</th>\n",
       "      <th>Home Ownership_Rent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>482087.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>26.3</td>\n",
       "      <td>685960.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>Short Term</td>\n",
       "      <td>99999999.0</td>\n",
       "      <td>47386.0</td>\n",
       "      <td>7914.0</td>\n",
       "      <td>749.0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1025487.0</td>\n",
       "      <td>10+ years</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>1181730.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>Long Term</td>\n",
       "      <td>264968.0</td>\n",
       "      <td>394972.0</td>\n",
       "      <td>18373.0</td>\n",
       "      <td>737.0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>751412.0</td>\n",
       "      <td>8 years</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1182434.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>Short Term</td>\n",
       "      <td>99999999.0</td>\n",
       "      <td>308389.0</td>\n",
       "      <td>13651.0</td>\n",
       "      <td>742.0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>805068.0</td>\n",
       "      <td>6 years</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>22.5</td>\n",
       "      <td>147400.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>Short Term</td>\n",
       "      <td>121396.0</td>\n",
       "      <td>95855.0</td>\n",
       "      <td>11338.0</td>\n",
       "      <td>694.0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>776264.0</td>\n",
       "      <td>8 years</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>13.6</td>\n",
       "      <td>385836.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>Short Term</td>\n",
       "      <td>125840.0</td>\n",
       "      <td>93309.0</td>\n",
       "      <td>7180.0</td>\n",
       "      <td>719.0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Annual Income Years in current job  Tax Liens  Number of Open Accounts  \\\n",
       "0   0       482087.0                  NaN        0.0                     11.0   \n",
       "1   1      1025487.0            10+ years        0.0                     15.0   \n",
       "2   2       751412.0              8 years        0.0                     11.0   \n",
       "3   3       805068.0              6 years        0.0                      8.0   \n",
       "4   4       776264.0              8 years        0.0                     13.0   \n",
       "\n",
       "   Years of Credit History  Maximum Open Credit  Number of Credit Problems  \\\n",
       "0                     26.3             685960.0                        1.0   \n",
       "1                     15.3            1181730.0                        0.0   \n",
       "2                     35.0            1182434.0                        0.0   \n",
       "3                     22.5             147400.0                        1.0   \n",
       "4                     13.6             385836.0                        1.0   \n",
       "\n",
       "   Months since last delinquent  Bankruptcies  ...        Term  \\\n",
       "0                           NaN           1.0  ...  Short Term   \n",
       "1                           NaN           0.0  ...   Long Term   \n",
       "2                           NaN           0.0  ...  Short Term   \n",
       "3                           NaN           1.0  ...  Short Term   \n",
       "4                           NaN           0.0  ...  Short Term   \n",
       "\n",
       "  Current Loan Amount  Current Credit Balance  Monthly Debt  Credit Score  \\\n",
       "0          99999999.0                 47386.0        7914.0         749.0   \n",
       "1            264968.0                394972.0       18373.0         737.0   \n",
       "2          99999999.0                308389.0       13651.0         742.0   \n",
       "3            121396.0                 95855.0       11338.0         694.0   \n",
       "4            125840.0                 93309.0        7180.0         719.0   \n",
       "\n",
       "   Credit Default  Home Ownership_Have Mortgage  Home Ownership_Home Mortgage  \\\n",
       "0               0                         False                         False   \n",
       "1               1                         False                         False   \n",
       "2               0                         False                          True   \n",
       "3               0                         False                         False   \n",
       "4               0                         False                         False   \n",
       "\n",
       "   Home Ownership_Own Home  Home Ownership_Rent  \n",
       "0                     True                False  \n",
       "1                     True                False  \n",
       "2                    False                False  \n",
       "3                     True                False  \n",
       "4                    False                 True  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       0\n",
       "1       0\n",
       "2       0\n",
       "3       0\n",
       "4       0\n",
       "       ..\n",
       "7495    0\n",
       "7496    0\n",
       "7497    0\n",
       "7498    0\n",
       "7499    0\n",
       "Name: Home Ownership_Have Mortgage, Length: 7500, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 可以看到上面独热编码后的数据是bool类型，试着转换为int类型，因为后续可能有的函数计算不支持bool值\n",
    "# 学习类型转换的方法\n",
    "data['Home Ownership_Have Mortgage'] =data ['Home Ownership_Have Mortgage'].astype(int)\n",
    "data['Home Ownership_Have Mortgage']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "到此为止，已经掌握了对离散变量做独热编码的所有方法\n",
    "1. 找到离散变量\n",
    "2. 独热编码映射\n",
    "3. 转换独热编码到int类型\n",
    "4. 填补每一列的缺失值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Id', 'Annual Income', 'Tax Liens', 'Number of Open Accounts',\n",
       "       'Years of Credit History', 'Maximum Open Credit',\n",
       "       'Number of Credit Problems', 'Months since last delinquent',\n",
       "       'Bankruptcies', 'Current Loan Amount', 'Current Credit Balance',\n",
       "       'Monthly Debt', 'Credit Score', 'Credit Default',\n",
       "       'Home Ownership_Home Mortgage', 'Home Ownership_Own Home',\n",
       "       'Home Ownership_Rent', 'Years in current job_10+ years',\n",
       "       'Years in current job_2 years', 'Years in current job_3 years',\n",
       "       'Years in current job_4 years', 'Years in current job_5 years',\n",
       "       'Years in current job_6 years', 'Years in current job_7 years',\n",
       "       'Years in current job_8 years', 'Years in current job_9 years',\n",
       "       'Years in current job_< 1 year', 'Purpose_buy a car',\n",
       "       'Purpose_buy house', 'Purpose_debt consolidation',\n",
       "       'Purpose_educational expenses', 'Purpose_home improvements',\n",
       "       'Purpose_major purchase', 'Purpose_medical bills', 'Purpose_moving',\n",
       "       'Purpose_other', 'Purpose_renewable energy', 'Purpose_small business',\n",
       "       'Purpose_take a trip', 'Purpose_vacation', 'Purpose_wedding',\n",
       "       'Term_Short Term'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 现在尝试结合之前的代码一次性对所有离散特征独热编码\n",
    "# 重新读取数据\n",
    "data = pd.read_csv(\"E:\\study\\PythonStudy\\python60-days-challenge-master\\data.csv\")\n",
    "# 找到离散变量\n",
    "discrete_lists = [] # 新建一个空列表，用于存放离散变量名\n",
    "for discrete_features in data.columns:\n",
    "    if data[discrete_features].dtype == 'object':\n",
    "        discrete_lists.append(discrete_features)\n",
    "\n",
    "# 离散变量独热编码\n",
    "data = pd.get_dummies(data, columns=discrete_lists, drop_first=True) \n",
    "\n",
    "data.columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此时还有个困难，如何找到所有独热编码后的新特征名呢？\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Home Ownership_Home Mortgage',\n",
       " 'Home Ownership_Own Home',\n",
       " 'Home Ownership_Rent',\n",
       " 'Years in current job_10+ years',\n",
       " 'Years in current job_2 years',\n",
       " 'Years in current job_3 years',\n",
       " 'Years in current job_4 years',\n",
       " 'Years in current job_5 years',\n",
       " 'Years in current job_6 years',\n",
       " 'Years in current job_7 years',\n",
       " 'Years in current job_8 years',\n",
       " 'Years in current job_9 years',\n",
       " 'Years in current job_< 1 year',\n",
       " 'Purpose_buy a car',\n",
       " 'Purpose_buy house',\n",
       " 'Purpose_debt consolidation',\n",
       " 'Purpose_educational expenses',\n",
       " 'Purpose_home improvements',\n",
       " 'Purpose_major purchase',\n",
       " 'Purpose_medical bills',\n",
       " 'Purpose_moving',\n",
       " 'Purpose_other',\n",
       " 'Purpose_renewable energy',\n",
       " 'Purpose_small business',\n",
       " 'Purpose_take a trip',\n",
       " 'Purpose_vacation',\n",
       " 'Purpose_wedding',\n",
       " 'Term_Short Term']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 对比独热编码前后的列名 即可\n",
    "data2 = pd.read_csv(\"E:\\study\\PythonStudy\\python60-days-challenge-master\\data.csv\")\n",
    "list_final = [] # 新建一个空列表，用于存放独热编码后新增的特征名\n",
    "for i in data.columns:\n",
    "    if i not in data2.columns:\n",
    "       list_final.append(i) # 这里打印出来的就是独热编码后的特征名\n",
    "list_final\n",
    "\n",
    "# 其实还可以通过data.columns.difference()方法来实现，请自行学习\n",
    "# 可以看到 想要实现一个结果有很多不同方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Annual Income</th>\n",
       "      <th>Tax Liens</th>\n",
       "      <th>Number of Open Accounts</th>\n",
       "      <th>Years of Credit History</th>\n",
       "      <th>Maximum Open Credit</th>\n",
       "      <th>Number of Credit Problems</th>\n",
       "      <th>Months since last delinquent</th>\n",
       "      <th>Bankruptcies</th>\n",
       "      <th>Current Loan Amount</th>\n",
       "      <th>...</th>\n",
       "      <th>Purpose_major purchase</th>\n",
       "      <th>Purpose_medical bills</th>\n",
       "      <th>Purpose_moving</th>\n",
       "      <th>Purpose_other</th>\n",
       "      <th>Purpose_renewable energy</th>\n",
       "      <th>Purpose_small business</th>\n",
       "      <th>Purpose_take a trip</th>\n",
       "      <th>Purpose_vacation</th>\n",
       "      <th>Purpose_wedding</th>\n",
       "      <th>Term_Short Term</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>482087.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>26.3</td>\n",
       "      <td>685960.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>99999999.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1025487.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>1181730.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>264968.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>751412.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1182434.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>99999999.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>805068.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>22.5</td>\n",
       "      <td>147400.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>121396.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>776264.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>13.6</td>\n",
       "      <td>385836.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>125840.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Annual Income  Tax Liens  Number of Open Accounts  \\\n",
       "0   0       482087.0        0.0                     11.0   \n",
       "1   1      1025487.0        0.0                     15.0   \n",
       "2   2       751412.0        0.0                     11.0   \n",
       "3   3       805068.0        0.0                      8.0   \n",
       "4   4       776264.0        0.0                     13.0   \n",
       "\n",
       "   Years of Credit History  Maximum Open Credit  Number of Credit Problems  \\\n",
       "0                     26.3             685960.0                        1.0   \n",
       "1                     15.3            1181730.0                        0.0   \n",
       "2                     35.0            1182434.0                        0.0   \n",
       "3                     22.5             147400.0                        1.0   \n",
       "4                     13.6             385836.0                        1.0   \n",
       "\n",
       "   Months since last delinquent  Bankruptcies  Current Loan Amount  ...  \\\n",
       "0                           NaN           1.0           99999999.0  ...   \n",
       "1                           NaN           0.0             264968.0  ...   \n",
       "2                           NaN           0.0           99999999.0  ...   \n",
       "3                           NaN           1.0             121396.0  ...   \n",
       "4                           NaN           0.0             125840.0  ...   \n",
       "\n",
       "   Purpose_major purchase  Purpose_medical bills  Purpose_moving  \\\n",
       "0                       0                      0               0   \n",
       "1                       0                      0               0   \n",
       "2                       0                      0               0   \n",
       "3                       0                      0               0   \n",
       "4                       0                      0               0   \n",
       "\n",
       "   Purpose_other  Purpose_renewable energy  Purpose_small business  \\\n",
       "0              0                         0                       0   \n",
       "1              0                         0                       0   \n",
       "2              0                         0                       0   \n",
       "3              0                         0                       0   \n",
       "4              0                         0                       0   \n",
       "\n",
       "   Purpose_take a trip  Purpose_vacation  Purpose_wedding  Term_Short Term  \n",
       "0                    0                 0                0                1  \n",
       "1                    0                 0                0                0  \n",
       "2                    0                 0                0                1  \n",
       "3                    0                 0                0                1  \n",
       "4                    0                 0                0                1  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 接着之前的，对bool特征进行类型转换\n",
    "for i in list_final:\n",
    "    data[i] = data[i].astype(int) # 这里的i就是独热编码后的特征名\n",
    "data.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                                  int64\n",
       "Annual Income                     float64\n",
       "Tax Liens                         float64\n",
       "Number of Open Accounts           float64\n",
       "Years of Credit History           float64\n",
       "Maximum Open Credit               float64\n",
       "Number of Credit Problems         float64\n",
       "Months since last delinquent      float64\n",
       "Bankruptcies                      float64\n",
       "Current Loan Amount               float64\n",
       "Current Credit Balance            float64\n",
       "Monthly Debt                      float64\n",
       "Credit Score                      float64\n",
       "Credit Default                      int64\n",
       "Home Ownership_Home Mortgage        int64\n",
       "Home Ownership_Own Home             int64\n",
       "Home Ownership_Rent                 int64\n",
       "Years in current job_10+ years      int64\n",
       "Years in current job_2 years        int64\n",
       "Years in current job_3 years        int64\n",
       "Years in current job_4 years        int64\n",
       "Years in current job_5 years        int64\n",
       "Years in current job_6 years        int64\n",
       "Years in current job_7 years        int64\n",
       "Years in current job_8 years        int64\n",
       "Years in current job_9 years        int64\n",
       "Years in current job_< 1 year       int64\n",
       "Purpose_buy a car                   int64\n",
       "Purpose_buy house                   int64\n",
       "Purpose_debt consolidation          int64\n",
       "Purpose_educational expenses        int64\n",
       "Purpose_home improvements           int64\n",
       "Purpose_major purchase              int64\n",
       "Purpose_medical bills               int64\n",
       "Purpose_moving                      int64\n",
       "Purpose_other                       int64\n",
       "Purpose_renewable energy            int64\n",
       "Purpose_small business              int64\n",
       "Purpose_take a trip                 int64\n",
       "Purpose_vacation                    int64\n",
       "Purpose_wedding                     int64\n",
       "Term_Short Term                     int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 填补每一列的缺失值\n",
    "data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                                   0\n",
       "Annual Income                     1557\n",
       "Tax Liens                            0\n",
       "Number of Open Accounts              0\n",
       "Years of Credit History              0\n",
       "Maximum Open Credit                  0\n",
       "Number of Credit Problems            0\n",
       "Months since last delinquent      4081\n",
       "Bankruptcies                        14\n",
       "Current Loan Amount                  0\n",
       "Current Credit Balance               0\n",
       "Monthly Debt                         0\n",
       "Credit Score                      1557\n",
       "Credit Default                       0\n",
       "Home Ownership_Home Mortgage         0\n",
       "Home Ownership_Own Home              0\n",
       "Home Ownership_Rent                  0\n",
       "Years in current job_10+ years       0\n",
       "Years in current job_2 years         0\n",
       "Years in current job_3 years         0\n",
       "Years in current job_4 years         0\n",
       "Years in current job_5 years         0\n",
       "Years in current job_6 years         0\n",
       "Years in current job_7 years         0\n",
       "Years in current job_8 years         0\n",
       "Years in current job_9 years         0\n",
       "Years in current job_< 1 year        0\n",
       "Purpose_buy a car                    0\n",
       "Purpose_buy house                    0\n",
       "Purpose_debt consolidation           0\n",
       "Purpose_educational expenses         0\n",
       "Purpose_home improvements            0\n",
       "Purpose_major purchase               0\n",
       "Purpose_medical bills                0\n",
       "Purpose_moving                       0\n",
       "Purpose_other                        0\n",
       "Purpose_renewable energy             0\n",
       "Purpose_small business               0\n",
       "Purpose_take a trip                  0\n",
       "Purpose_vacation                     0\n",
       "Purpose_wedding                      0\n",
       "Term_Short Term                      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum() # 统计每一列的缺失值个数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                                0\n",
       "Annual Income                     0\n",
       "Tax Liens                         0\n",
       "Number of Open Accounts           0\n",
       "Years of Credit History           0\n",
       "Maximum Open Credit               0\n",
       "Number of Credit Problems         0\n",
       "Months since last delinquent      0\n",
       "Bankruptcies                      0\n",
       "Current Loan Amount               0\n",
       "Current Credit Balance            0\n",
       "Monthly Debt                      0\n",
       "Credit Score                      0\n",
       "Credit Default                    0\n",
       "Home Ownership_Home Mortgage      0\n",
       "Home Ownership_Own Home           0\n",
       "Home Ownership_Rent               0\n",
       "Years in current job_10+ years    0\n",
       "Years in current job_2 years      0\n",
       "Years in current job_3 years      0\n",
       "Years in current job_4 years      0\n",
       "Years in current job_5 years      0\n",
       "Years in current job_6 years      0\n",
       "Years in current job_7 years      0\n",
       "Years in current job_8 years      0\n",
       "Years in current job_9 years      0\n",
       "Years in current job_< 1 year     0\n",
       "Purpose_buy a car                 0\n",
       "Purpose_buy house                 0\n",
       "Purpose_debt consolidation        0\n",
       "Purpose_educational expenses      0\n",
       "Purpose_home improvements         0\n",
       "Purpose_major purchase            0\n",
       "Purpose_medical bills             0\n",
       "Purpose_moving                    0\n",
       "Purpose_other                     0\n",
       "Purpose_renewable energy          0\n",
       "Purpose_small business            0\n",
       "Purpose_take a trip               0\n",
       "Purpose_vacation                  0\n",
       "Purpose_wedding                   0\n",
       "Term_Short Term                   0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 用均值填补\n",
    "# 循环遍历这个列表中的每一列\n",
    "for i in data.columns:\n",
    "    if data[i].isnull().sum() > 0: # 找到存在缺失值的列\n",
    "        #计算该列的均值\n",
    "        mean_value = data[i].mean()\n",
    "        #用均值填充缺失值\n",
    "        data[i].fillna(mean_value, inplace=True)\n",
    "\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "刚才这段代码中出现了一个警告，我们昨天说到了这个警告没什么用，但是会影响你的观感。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')  # 忽略警告信息"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在在py文件中 一次性处理data数据中所有的连续变量和离散变量\n",
    "1. 读取data数据\n",
    "2. 对离散变量进行one-hot编码\n",
    "3. 对独热编码后的变量转化为int类型\n",
    "4. 对所有缺失值进行填充"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "那么这样有没有问题呢？实际上是存在逻辑问题的\n",
    "\n",
    "原则上应该先填补缺失值再独热编码，如果顺序颠倒的话，用众数补全，比如北京上海深圳001，有可能三个数的众数都是0，就会变成000。\n",
    "\n",
    "那么对于object对象如何填补缺失值呢？大家尝试自己借助AI来实现\n",
    "\n",
    "这里作为课后作业"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外，大家其实非常喜欢问我，预处理、描述性统计、特征筛选等等流程谁先谁后，实际上没有严格的顺序，你自己要能把握住主线，ai经常在这里犯错。\n",
    "\n",
    "当你指出ai的错误的时候，他会告诉你你说的对。----希望大家也很快出现这一时刻.\n",
    "\n",
    "来点人生哲理小课堂\n",
    "\n",
    "一定要实践，不要丧失自己的主体性，任何丧失主体性的行为最后一定是南辕北辙。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "vs",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
